Papers with Ordinary Differential Equations
ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation (2022.acl-long)
Copied to clipboard
Bei Li, Quan Du, Tao Zhou, Yi Jing, Shuhan Zhou, Xin Zeng, Tong Xiao, JingBo Zhu, Xuebo Liu, Min Zhang
| Challenge: | Residual networks are an Euler discretization of solutions to Ordinary Differential Equations (ODE). |
| Approach: | They propose a residual block of layers in Transformer that can be described as a higher-order solution to ODE. |
| Outcome: | The proposed architecture can gain large improvements over strong baselines at a slight cost in inference efficiency. |
Knowledge Graph Representation Learning using Ordinary Differential Equations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge Graph Embeddings (KGEs) map entities and relations from knowledge graphs into a geometric space. |
| Approach: | They propose a neuro differential KGE that embeds nodes of a KG on the trajectories of Ordinary Differential Equations (ODEs) they represent each relation (edge) in a knowledge graph as a vector field on several manifolds. |
| Outcome: | The proposed model can preserve graph characteristics including structural aspects and semantics and avoid wrong inferences. |
IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained Language Models (PLMs) have shown remarkable performance on diverse NLP tasks through pre-training and fine-tuning. |
| Approach: | They propose a numerically robust IM-connection incorporating a layer of BERT as a solution of Ordinary Differential Equations (ODEs) . Experimental results validate the robustness of IM BERT under various conditions. |
| Outcome: | The proposed model outperforms the existing model on the adversarial GLUE dataset by 5.9%p on low-resource scenarios. |